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(54) Title: VEHICLE VISION SYSTEM 

(57) Abstract 

A vehicle viewing system including a camera 
system for generating a signal corresponding to a scene 
by integrating light from the scene incident on pixel 
cells having a variable integration time, a display 
system for presenting a visual representation of tne 
scene, and a processor system operable to determine 
the camera system integration time based on ""ghttos 
levels in the scene. The camera »ystm preferaWy 
includes an optical array of photogate CMOS active 
pixel sensor cells and support electronics implementing 
a plurality of integration and processing architectures. 
The camera system may seclude dig^output 
and an input amr-^-1 filter to hmtt light striking 
the optical array. The processor system includes an 
image brightness detector to determine measures of 
overall image brightness and a display pixel hmunance 
mapping control to determine luminance settings Tor 
the display system. The image brightness detectormay 
include circuitry that produces a brightness histogram. 
The processor system may determine the intenrty of 
the display system based on the brightness of tne 
scene, ambient light levels, and glare on the display. 
The display system includes a display and a display 
attenuation filter for limiting the intensity as viewed 



by the operator. 
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VEHICLE VISION SYSTEM 

Technical Field 

The present invention relates to a vision 
syscem for vehicles . 

5 Background Art 

Viewing systems are important to the proper 
operation of motor vehicles. Such systems must be 
capable of operating in a wide range of inter- scene 
brightness ranging from strong sunlight to moonlight. 
10 They must also perform well in a wide range of intra- 
scene brightness as may occur from viewing a nighttime 
scene containing car headlights. Still further, frame 
rates must be attainable so that the displayed view 
appears to be real-time. 

15 Traditionally, vehicle viewing systems have 

used a plurality of mirrors to allow the operator to 
see objects behind and beside the vehicle. The use of 
mirrors has resulted in several difficulties, includ- 
ing blind spots due to opaque body structures, distor- 

20 tion due to convexity, inability to modify color bal- 
ance for night vision, and wind resistance due to 
outside mounting. 

Another difficulty with current vehicle 
viewing systems is an inability to properly display 
25 the intensity of a dimly lit scene. The human eye is 
much more sensitive to light when adapted to dimly 
lighted street and highway driving at night than in 
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the daytime. Viewing relatively bright images, such 
as a display screen with a relatively high intensity, 
causes changes in the retina which substantially and 
subtly reduce visual sensitivity to dimly lit scenes. 
5 Following exposure to brighter lights, this effect 

reduces the perception of and resulting reaction time 
co dimly lit scenes. 

Still another' difficulty with current vehi- 
cle viewing systems is the inability to alter the 

10 balance between dim and bright portion of a scene. 

For example, headlamps of a trailing .vehicle are fre- 
quently annoying to a c driver. The brightness obscures 
visibility of much or all of other features in the 
rearward field of view, and may be bright enough to . 

15 further mask visibility in portions of the forward 

field of view. Electrochromic mirrors have eliminated 
most of the annoyance glare without darkening 
excesively but cannot alter the balance between dim 
and bright portions of the same scene and cannot 

20 effectively amplify the intensity of a dimly lit 
scene . 

Yet another difficulty with current vehicle 
viewing systems is the inability to shift the color 
balance of a displayed scene in response to changing 
25 ambient light conditions. In particular, blue light 
is more damaging to night vision than red light. 

To reduce these problems and others, vehicle 
viewing systems employing cameras and displays have 
been proposed. Typically, one or more cameras are 
30 mounted on the sides, roof or rear of the vehicle. 

The video output may be processed to bl nd the views 
from multipl cameras, extract distance information, 
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and adapt to changing environmental conditions. One 
or more displays are used to present the raw or pro- 
cessed video data to the operator. 

Consequently, a need exists to produce a 
vehicle viewing system with extended dynamic range 
capable of operating effectively across wide intra- 
scene and inter-scene brightness levels. The system 
muse also reduce the effects of display glare and not 
subject the vehicle operator to light levels and color 
balances that might compromise night vision. 



Summary Of The Invention 

It is a primary object of the present inven- 
tion to provide a system and method for viewing scenes 
around a vehicle in a variety of environmental condi- 
tions . 

Another object of the present invention is 
to provide a system and method of displaying an image 
of a scene with minimal exposure of the driver to high 
display light levels and glare which may impair night 
vision. 

Another object of the present invention is 
to provide a system and method of displaying an image 
of a scene with appropriate color balance to prevent 
compromising the night vision of a driver. 

Another object of the present invention is 
to provide a system and method for viewing details in 
scenes that may be obscured due to dim lighting or 
masked by bright light sources. 
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Still another object of the present inven- 
tion is to provide a system and method for viewing 
scenes with wide inter-scene brightness levels. 

A further object of the present invention is 
5 to provide a system and method for viewing scenes with 
wide intra-scene brightness levels. 

In carrying out the above objects and other 
objects and features of the present invention, a 
system is provided that includes a camera system to 

10 generate a signal corresponding to the scene by 

integrating light from the scene incident on pixel 
cells having a variable integration time, a display 
system to present a visual representation of the 
scene, and a processor system to determine the camera 

15 system integration time based on brightness levels in 
the scene . 

In an embodiment , the camera system includes 
an optical sensor, a lens system for focusing an Mmage 
of the scene onto the optical sensor, and a variable 

20 input attenuation filter for litniting the light from 
the image which strikes the optical sensor. The 
optical sensor is preferably comprised of an array of 
photogate CMOS active pixel sensors cells. Support 
eleGjtronics ^low fj?r, a plurality of architectures 

25 including double integration, dual integration, inter- 
lacing, and individual cell pixel reset. The variable 
input attenuation filter may be implemented with an 
electrochromic window. The camera system may include 
a digitized output. 



30 



In an embodiment, the processor system is 
operable to determine camera and display system set- 
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tings. The processor system may include an image 
brightness detector to determine measures 1 of overall 
image brightness and a display pixel luminance mapping 
control to determine luminance settings for the 
5 display system. The processor may be further operable 
:o accept inputs from ambient light sensors, direct 
glare sensors, and manual intensity adjustments for 
determining display intensity. 

In an embodiment, the display system 
10 includes a display and a display variable attenuation 
filter for limiting the light emitted from the 
display. The display variable attenuation filter may 
be implemented with an electrochromic window. 

In an illustrative embodiment, the image 
15 brightness detector determines a histogram of 
brightness values in image frames. 

In another illustrative embodiment, the 
display pixel luminance mapping control is implemented 
with a memory accepting an address formed from an 
20 image brightness histogram and individual image pixel 
brightness values and producing a corresponding 
display pixel luminance value. 

The above objects and other objects, fea- 
tures, and advantages of the present invention are 
25 readily apparent from the following detailed descrip- 
tion of the best mode for carrying out the invention 
when taken in connection with the accompanying draw- 
ings. 
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Brief Description Of The Drawings 



FIGURE 1 is a general schematic diagram of a 
vehicle including the present invention; 

FIGURE 2 is a block diagram of a preferred 
embodiment according to the present invention; 

FIGURE 3 is a schematic diagram for 
implementing an image brightness detector according to 
the present invention; 

FIGURE 4 is a schematic diagram for 
implementing a display pixel luminance mapping control 
according to the present invention; 

FIGURE 5 is a graph of display luminance 
mapping according to the present invention; 

FIGURE 6 is a graph of control 
characteristics 'for variable attenuation filters 
according to the present invention; and 

FIGURE 7 is a graph of display attenuation 
ratio versus ambient light conditions according to the 
present invent ion . 

Referring now to Figure 1, a schematic dia- 
gram of the present invention is shown. Vehicle 20 is 
driven by operator 22. One or more camera systems 26 
are operative to view a scene 24- In the example 
shown, scene 24 is generally behind vehicle 20. Of 
course, however, camera system 26 may be oriented in a 
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variecy of ways to view scenes at other locations 
about vehicle 20, including but not limited to, the 
sides, back, front, bottom, top, and inside. In the 
example shown, signals representative of the scene are 
sent via channel 28 to a processor system 30. Input 
from an ambient light sensor 34 and direct glare sen- 
sor 36 are also available to processor system 30. 
Processor system 3 0 produces an enhanced image of 
scene 24 on one or more display systems 32. 

In a particularly useful embodiment, a 
conventional mirror system is augmented by camera 
systems 26 which cover a wide field of view to the 
back and sides so that pedestrians or other objects 
directly back of vehicle 20 may be seen and so than 
oncoming traffic from the sides may be seen. The 
system is designed so that, when backing out of a 
parking spot, oncoming vehicles may be seen before 
backing into the lane of travel. This requires camera 
system 26 with a near 180 degree field of view or 
several camera systems 26 mounted near the rear of the 
vehicle. An analogous system with a camera or cameras 
mounted near the front of the vehicle is adapted to 
view cross traffic at a "blind" intersection before 
entering the lane of travel of the cross traffic. 
These are desirable applications for the present 
invention which supplement the viewing function of 
conventional rearview mirrors. 

Referring now to Figure 2, a block diagram 
of a preferred embodiment according to the present 
invention is shown. 



WO 99/33684 



PCT/US98/27654 



-8- 

Camera Syptam 

Camera system 26 accepts image rays 50 from 
scene 24 . Image rays 50 pass through optional input 
variable attenuation filter 52 emerging as attenuated 
5 image rays 54. Rays 50 or 54 are focused by lens 

system 56, becoming focused rays 58. An image sensor 
array 60 is placed in the focal plane of lens system 
56 . The image sensor array is comprised of individual 
pixel sensors, ideally arranged in rows and columns. 

10 An image sensor interface and control unit 62 provides 
control signals 64 to image sensor array 60 and re- 
ceives electrical signals 66 corresponding to scene 24 
from image sensor array 60. Image sensor interface 
and control 62 may operate on signals 66, preferably. 

15 including digitizing the signals, prior to forwarding 
the camera system output signals 68. If optional lens 
attenuating filter 52 is used, the amount of 
attenuation is controlled by lens attenuation control 
70 through lens attenuation filter signal 72. 

20 In a preferred embodiment, camera system 26 

is designed to handle a large dynamic range. A sig- 
nificant improvement over prior systems is the ability 
of camera system 26 to capture and transmit detail in 
scene 24 that has previously been obscured due to low 

25 illumination levels or due to glare from lights such 



One limitation on the dynamic range of cam- 
era system output is due to the pixel sensors in the 
image sensor array. A preferr d embodiment uses com- 
30 plimentary metal-oxide semiconductor/metal -on- silicon 
(CMOS) photogate active pixel sensor (APS) cells. The 
photogate in each cell is used to integrate charge 
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developed from incident light. A storage site is 
capable of holding the integrated charge. 1 The storage 
site can be reset to a reference level indicative of 
pixel sensor noise. A selectable buffer circuit 

5 outputs a signal proportional to the integrated charge 
or reference value at the storage site. By 
subtracting the reference noise signal from the 
integrated charge signal, a significant effect of the 
noise can be eliminated, increasing pixel sensor 

10 sensitivity. 

Another limitation on the dynamic range of 
camera systems has been the restriction on the length 
of time for integrating charge produced by incident 
light. Current systems limit the integration time to 

15 slightly less than the frame time. Since it is desir- 
able to display scene 24 as near real-time, a high 
frame rate of preferably not less than 3 0 frames per 
second is required. Traditionally, this has resulted 
in integration times of no greater than 33 

20 mi 1 1 i seconds . 

An optical sensor combining image sensor 
array 60 and image sensor interface and control 62 
with a wide dynamic range is described in pending U.S. 
Patent Application Serial No. 09/002,400 titled "Wide 

25 Dynamic Range Optical Sensor" to Stam et. al. and is 
hereby incorporated by reference. One method de- 
scribed implements a double integration architecture 
by using storage in each pixel cell to hold the charge 
integrated from the previous frame period while inte- 

30 grating charge for the current frame period. By 

summing the signals representing charge integrated 
during the current and previous frame periods, a 
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signal is produced thac has an effective integration 
time of twice the frame period. 

A second method described by Stam et. al . 
uses an interlacing architecture to read a subset of 
5 pixel sensors each frame period. Those pixel sensors 
not read in a given frame period continue to integrate 
light -induced charge. By reading each pixel sensor ac 
a period that is a multiple of the frame time, an 
effective integration time greater than the frame time 
10 is achieved. Values representing pixel cells not read 
in a given frame period may be interpolated from 
neighboring pixel cells which are read. 

A third method described by Stam et . al . 
uses a dual integration architecture to provide two 
integration signals. A first signal is generated by 
integrating charge over a relatively long period. 
This charge is then stored in the pixel sensor cell. 
A second signal is generated by integrating charge 
over a relatively short period. If a signal 
corresponding to the long integration period is less 
than a threshold, the long integration signal is used 
as the output. If the long integration signal is not 
less than a threshold, the short integration signal is 
used. This provides an output signal with greater 
resolution at low light levels while still covering a 

A fourth method described by Stam et. al. 
uses individual pixel resets to extend the effective 
dynamic range. In this method, individual or groups 
30 of pixel sensors can be reset during the integration 
time, thereby providing a shorter integration period. 
Areas of the image of scene 24 which are dimly lit 
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receive longer integration periods than areas which 
are brightly lit. Using this technique, the bright 
headlamps of a trailing vehicle in scene 24 are 
locally sensed at a much lower sensitivity so that 

5 saturation in the corresponding portion of the image 
is reduced or eliminated while still sensing the 
remainder of scene 24 with sufficient sensitivity. 
This allows viewing of detail which has heretofore 
been obscured by bright headlamps. Since there are 

10 frequently only two headlamps causing most of the 

glare in the field of view of scene 24 and since che 
image of each headlamp projects a small sized area 
cypically striking at most a two-by-two pixel sensor 
area, the capability to position as few as two reduced 

15 integration time windows, of standard size, such as, 

for example, three-by- three pixel sensors, can yield- 
considerable benefit in control of brightness in the 
image of scene 24 . 

In a more sophisticated embodiment, addi- 
20 tional perhaps variable sized reset windows can be 

used to reduce the integration times for portions of 
image 24 corresponding to reflection and glare. The 
ability to dynamically adjust reset window position 
and reset time will be provided. The method of con- 
25 trolling the reset period for a pixel sensor is de- 
scribed in "Image. Sensors With Individual Pixel Re- 
set," page 34 of NASA Tech Brief NPO-1973 of November 
1996 by pecht, Pain and Possum. 

The four methods described in Stam et . al. 
30 can be implemented in the same optical sensor. 

Control inputs determine which methods are in use, the 
integration times, and the res t times. 
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Even with an image sensor arra^ 60 
possessing a relatively large intra-scene dynamic 
range, the ambient light level over which camera 
system 2 6 must operate is likely to be greater than 
5 can be accomplished solely with the image sensor 60. 
For example, there should be some visibility even in 
bright moonlight and the ratio between the intensify 
cf bright sunlight and bright moonlight is roughly 
1,000,00*0 to one. Thus, there must be one or more 

10 methods to increase the range over which image sensor 
6 0 can be used well beyond that provided by the wide 
intra-scene dynamic range of image sensor 60. To 
accommodate such wide inter-scene variations in 
lighting conditions, variable attenuation filter 52 

15 can be used. In one embodiment, a lens with an 

automatic variable iris is used. However, such mecha- 
nisms are presently expensive and a lens system 56 
adapted to work with an iris is likely to require more 
elements, working against the requirement for 

20 exceptionally high contrast and low light dispersion 
required to -make the large dynamic range pqssible as 
described below. It is thus preferred to use a lens 
which has a fixed aperture. Input attenuation filter 
52 may also be implemented as an electrically 

25 controlled shutter. 



In the preferred embodiment, input 

trochromic window. The window transitions from "sub- 
stantially clear to maximum attenuation based on at- 
30 tenuation filter signal 72. The steady state attenua- 
tion is a reasonably stable and reproducible function 
of voltage so that, having experimentally determined 
the relationship between voltage and light attenua- 
tion, a controller may be used to set the amount of 
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attenuation. This allows camera system 26 to employ a 
highly sensitive image sensor array 60 without exces- 
sive saturation in bright daylight. The use of an 
electrochromic window for implementing filter 52 is 
described with regards co Figure 6 below. 

A digital outpuc from camera system 26 is 
desired. Image sensor interface and control 62 may 
use an II- or 12-bit analog-to digital converter (ADC) 
to read the pixel output which indicate the respective 
integrated light level received at the pixel sensor 
sites . 

An alternative embodiment to the above ADC 
is a multi-range ADC having fewer bits. A dual or 
multiple ranging scheme may also be used including .a 
digitized value and a range indication. For example, 
a dual integration architecture may be implemented in 
image sensor array 60. Two different integration 
periods where one may be, for example, eight times as 
long as the other, may be used. For each pixel, if 
the reading with the longer integration is in a range 
which would not cause a full scale ADC readout value, 
it is digitized with, for example, an 8 bit converter. 
Nine bits consisting of the 8-bit digitized value 
along with a range indication bit set to indicate the 
longer integration period are provided as camera sys- 
tem output signal 68. Otherwise, when the light level 
at the pixel sensor is too high for the long integra- 
tion period, the reading for the short integration 
period is digitized and this 8 -bit value along with a 
range indication bit set to indicate the short inte- 
gration period are output as signal 68. Particularly 
in this configuration, the values each consisting of a 
magnitude and a range indicator are not in a good form 
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to send to display system 32 and in either this con- 
figuration or the 11- or 12-bit ADC, the total bright- 
ness range will be more than 2000 to one if linearly 
reproduced on a display. 

A further non-linear ADC embodiment utilizes 
a logarithmic preamplifier or logarithmic converter to 
provide a greater density of quantization levels at 
low light levels than at high light levels. 
< 

Camera system 2 6 with a wide dynamic range 
places strict requirements on lens system 56. Some 
percentage of the light which emanates from a bright 
portion of scene 24 strikes lens system 56 and is • 
scattered by the optical surfaces of the lens. Some 
percentage of this scattered light, either directly of 
through further interactions with components in camera 
system 26 ends up striking image sensor array 60 in an 
otherwise dimly lighted portion of the image. Another 
portion of the light which strikes a refracting or 
diffracting optical surface is reflected Off of the 
surface. In many cases, this is an unwanted reflec- 
tion and a portion of this light; may also end up 
striking an otherwise dimly lighted portion of the 
image. The problem is exacerbated by the need for 
high low- light sensitivity which necessitates the use 
of a relatively fast lens, for example one with an f 

range of f 1 . 

To perform well, lens system 56 needs to 
have as few optical surfaces as practical and the 
refracting surfaces must be coated with anti- 
reflection coatings. The requirement to minimize the 
number of optical surfaces suggest the choice of a 
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lens with one or more aspheric surfaces. t Since 
scattering is a problem with diffractive "optics, the 
requirement suggests that the lens choice should be 
limited to one with refracting or reflecting and not 
diffracting optical elements. 

Processor System 



Referring again to Figure 2, processor sys- 
tem 3 0 is further described. The camera system outpuc 
68 is processed by image brightness detector 74 and 
display pixel luminance mapping control 76. Image 
brightness detector 74 may determine the brightness 
level of the entire image and may determine brightness 
levels of regions within the image-. Display pixel 
luminance mapping control' 76 may compress the wide 
dynamic range of camera system output 68 to one which 
is comfortably viewable by operator 22. Display pixel 
luminance mapping control 76 may also increase the 
visibility of scene 24 while limiting higher light 
levels which are detrimental to the night vision of 
operator 22. Display luminance signal 78 is processed 
by display interface 80 to produce display signal 82 
for display system 32. Control' logic 84 is in commu- 
nication with image brightness detector 74 through bus 
86, display pixel luminance mapping control 76 through 
bus 88, display interface 80 through bus 90, image 
sensor interface and control 62 using image sensor 
control signal 92, input attenuation control 70 using 
input attenuation control signal 94, and other ele- 
ments as will be described forthwith. 

In image brightness detector 74, camera 
system output signal 68 is sampled to obtain digitized 
pixel readings. From thes samples, control logic 84 
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computes and frequently updates the average pixel 
brightness in the frame and also updates the number of 
pixels which are at maximum brightness and minimum 
brightness in an image frame . Control logic 84 may 
5 periodically send control signals 92 to image sensor 
interface and control 62 to adjust the integration 
cime so that the desired average image brightness in 
camera system output signal 68 is maintained. In 
another embodiment, the standard deviation of the 
10 brightness in camera system output signal 68 over a 
frame can be computed. 

In a further embodiment, the integration 
period and the resulting average brightness are de- 
creased when too high a percentage of pixels are at 

15 their maximum brightness level . Additionally, when 
few pixels are saturated but a larger percentage are 
at minimum brightness, the integration period is in- 
creased to raise the average brightness. When scene 
. 24 is too bright to adequately control the overall 

20 brightness of camera output signal 68 using integra- 
tion period alone, input variable attenuation filter 
52 is darkened using input attexination filter signal 
72 to provide the desired degree of added attenuation. 
An illustrative embodiment of image brightness detec- 

25 tor 74 is described with regards to Figure 3 below. 

^u^^ia^. „ ^esifigj^ <s6nt5^©iied^ o^mMM^M^^ , 

range of brightness primarily by changing the integra- 
tion time and method in order to reasonably center the 
image exposure in the electrical readout range of the 
30 pixel sensors and associated electronics. This bal- 
ance is approximately maintained through display pixel 
luminance mapping control 76 so that, without further 
adjustments, th average display brightness will re- 
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main nearly constant. This is not sufficient to con- 
trol the intensity of display system 32, however, 
since the display must be much brighter to be seen in 
the day than in the night. Furthermore, in spite of 
compression and the effective limiting of the dis- 
played level of brighter headlights, the displayed 
image of scene 24 sciil has a large dynamic range 
which may, for example, be 200-to-one. For operator 
22 to continue co see this range of brightness without 
being far too bright at night, the average intensity 
of display system 32 will have to be adjusted over a 
very large range and the adjustment will have to be 
well characterized to what is required. Any system 
which provides only two settings such as, for example, 
a high intensity level with headlamps off and a low 
intensity level with headlamps on will on, will be 
highly inadequate. An important consideration in 
control of the intensity of display system 32, 
particularly in low ambient light conditions, is that 
the maximum and average intensities should be main- 
tained at levels which are generally as low as reason- 
able to convey the required information so that the 
subsequent ability of operator 22 to discern and re- 
spond to dimly lit images is not unnecessarily compro- 
mised . 

An embodiment has the intensity of camera 
system 26 adjusted in inverse proportion to the camera 
sensitivity setting. The integration time calculated 
in processor system 30 forms the basis for determining 
the brightness setting. A lookup procedure can then 
be used to convert the integration time to a bright- 
ness setting based on display type, display mounting 
relative to operator 22, vehicle 20 lighting condi- 
tions, and other factors. A modification would use 
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averages of integration times to stabilize brightness 
settings . 

Another embodiment has the intensity of 
display system 32 leveling off to a minimum threshold 
5 at approximately the light level for which the inte- 
gration period is at a maximum (i.e. camera system 26 
is at maximum sensitivity). Under these conditions, 
scene 24 is likely to be dimmer than operator 22 can 
see in a normal mirror so that the displayed image 
10 must be enhanced over levels which would otherwise 

approximate the brightness of the scene being imaged. 

Still another embodiment can be used when 
the lights of a trailing vehicle are adding signifi- 
cantly to the average rearward light level. Camera' 

15 system 26 will be adjusted for lower sensitivity and, 
under the method of the first improvement, display 
system 32 will be set to a higher intensity level as a 
result. This higher intensity level may be too high 
for the forward ambient light level to which the eyes 

20 of operator 22 have become adjusted. To compensate, a 
second average light level is calculated omitting the 
values from brighter pixels. The second average is 
compared to a first average of all pixels and, if the 
second average is substantially lower than the first 

25 average, the display intensity may be reduced to cor- 
respond more nearly to the level obtained when the 
bright light sources are not included. 

Still another embodiment for controlling the 
intensity of display system 32 uses a non-linear ap- 
30 proach based on output from camera system 26 with a 

dual integration architecture. Control logic 84 forms 
a number from the data value and range (short or long 
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integration time) indication. This number is used as 
an index into a lookup table to obtain the display 
intensity setting. The magnitude of the intensity 
output for the condition where strong brightness is 
present should be an approximately logarithmic func- 
tion of the magnitude of camera system output signal 
S3 brightness. This embodiment is further described 
with regards to Figures 4 and 5 below. 

Still another embodiment for controlling the 
intensity of display system 32 uses frame-based image 
processing. Various regions of a frame are examined 
and the local intensity is adjusted based on localized 
spacial characteristics of the scene. For example, 
brightness levels in brighter zones may be scaled 
down. Also, areas surrounding bright lights might be 
compressed differently and more severely than other 
areas in the image. Also, if an analysis show the 
lighting to be very flat, particularly when headlamp 
glare is not present, the compression may be elimi- 
nated or brightness expansion may be used to increase 
contrast and help definition of detail. 

Yet another embodiment for controlling the 
intensity of display system 32 uses ambient light 
signal 96 from forward facing ambient light sensor 34. 
The eyes of operator 22 are adapted mainly to the 
average light level within a generally forward facing 
field of view. A time average of ambient light signal 
96 may be used to provide an indication of the ambient 
level seen by operator 22. Ambient light signal 96 
may be used in place of or in addition to sensitivity 
settings of camera system 26 to program the average 
intensity of display system 32 between a minimum 
threshold at low ambient light levels and a high 
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threshold for high ambient light levels. The use of 
forward facing ambient light sensors are described in 
U.S. Patent 4,917,477, titled "Automatic Rearview 
Mirror System for Automotive Vehicles" to Bechtel et . 
5 al . and hereby incorporated by reference. 

A further embodiment for controlling the 
intensity of display system 32 uses. glare signal 98 
f rem direct glare sensor 36. Direct glare sensor 36 
is placed so as to sense light levels falling on dis- 
10 play system 32 which may be excessive relative to the 
prevailing ambient light condition. The intensity of 
display system 32 may be increased form the otherwise 
normal level when these conditions are present to 
prevent wash-out . 

15 A still further embodiment does not require 

extensive calculations and may be used as a stimuli to 
alter the intensity of display system 32. When 
proportions of saturated and dark pixels are both 
small, an image of lower contrast is indicated and a 

20 lower degree of compression or expansion may be used. 

Yet a further embodiment for stimulating the 
modification of display system 32 intensity is to 
estimate or determine the standard deviation of 
brightness over the image frame. 

embodiments for controlling the intensity of display 
system 32 are incorporated. 

Image brightness detector 74, display pixel 
luminance mapping control 76, and control logic 84 are 
30 closely related. Either or both of detector 74 and 
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luminance mapping control 76 may be partially or com- 
pletely merged into control logic 84. Further control 
logic 84 may modify camera system output 68 prior to 
use in either detector 74 or luminance mapping control 
76. This modification could include filtering and 
feature extraction. 



Display System 



Referring again to Figure 2, display system 
3 2 is further described. Display signal 82 from dis- 
play interface 80 is fed to display 100. Display rays 
104, generated by display 100, pass through optional 
display variable attenuation filter 106 and emerge as 
filtered display rays 108.. Filtered display rays 103 
representing scene 24 are viewed by operator 22. If-- 
optional display attenuating filter 100 is used, the 
amount of attenuation is controlled by display attenu- 
ation control 110 through display attenuation filter 
signal 112 . 

Display 100 may be implemented with a number 
of different technologies, including cathode ray tubes 
(CRT) , field emission displays, and back lit liquid 
crystal displays (LCD) . Display 100 is preferably 
full color but, due to current cost constraints and 
difficulties with color displays at low brightness 
levels, a monochrome display may be used. As a com- 
promise, a two-color display designed to appear rea- 
sonably natural while allowing operator 22 to 
distinguish red tail lamps from head lamps or blue 
emergency signals from amber turn signal lamps may be 
used. 
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Many displays do not have the brightness 
range necessary to cover a good dynamic range within 
one frame in addition to a very large total brightness 
range for use in ambient light conditions which range 
5 from darkness to bright sunlight. To reduce the re- 
quirements on display 100, a display variable attenua- 
tion filter 106 may be used. In the preferred embodi- 
ment, attenuation filter 106 is implemented with an 
eiectrochromic window. - The attenuation filter is 
10 controlled by processor system 30 through display 

attenuation control signal 114. A method for control- 
ling filter 106 is described with regards to Figures 6 
and 7 below. 

Control of the intensity of display system. 
15 3 2 may be done solely with display attenuation filter 
106, with control of display 100 brightness or, in the 
preferred embodiment, with a combination of both tech- 
niques . 

To compensate for variations in preferences 
20 of individual operators, a manual brightness adjust- 
ment 116 can be included. Manual brightness signal 
118 is used by processor system 3 0 to modify calculat- 
ed brightness levels. Optionally, a brightness con- 
trol built into display 100 may be used as a supple - 
25 ment or alternate to display pixel luminance mapping 
— ^g*^ 

justment is likely to. still be required to meet the 
widely varying requirements of vehicle ambient light- 
ing conditions. 

30 In addition to controlling the brightness of 

rays 102 from display system 32 , it is desirable to 
control the color in low- light conditions. Studies 
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have indicated that blue light is more disruptive than 
red light to human night vision. If display 100 has 
full cr partial color, it may be advantageous to modi- 
fy the color balance in rays 104 observed by operator 
22 in low ambient light conditions. One method is to 
vary the color balance of display 100 so as to shift 
displayed color away from the shorter blue wave- 
lengths . Another method is co provide a blue blocking 
filter in display variable attenuation panel filter 
106 such that, as the amount of attenuation increases 
in filter 106, the shorter wavelength visible light is 
attenuated to a greater extent than longer wavelength 
visible light. Both methods may be implemented in the 
same system. 

The system described in Figure 2 may be 
implemented as hardware, software, or a combination of 
both. Signal paths may be implemented as discrete 
wiring, optical cabling, buses, and other channels and 
mediums as is well known in the art. Buses may be 
implemented as serial or parallel connections, and 
various buses may be combined. Furthermore, elements 
described may be combined or further partitioned 
within the spirit and scope of this invention. 

T m*9 m B^^ ht " nflBB Detector 

Referring now to Figure 3, a schematic 
diagram for implementing image brightness detector 74 
according to an embodiment of the present invention is 
shown. For this illustrative embodiment, camera 
system 26 implements a dual integration architecture 
with an eight -to-one ratio between long and short 
integration times. Light incident on each pixel 
sensor is represented by an 8 -bit illuminance value 
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plus an additional bit indicating whether long or 
short integration was used. Effectively, this results 
in an 11 -bit illuminance value for which approximately 
one-quarter of the possible 2048 numbers can be pro- 
5 duced. Image sensor array 60 has a 320 by 240 pixel 
sensor array. For this illustrative embodiment, dis- 
play 100 accepts an 8 -bit luminance value. 

The circuit described generates a histogram 
which includes the number of saturated pixel ceils, 

10 the number of dark pixel cells, and the number of 

pixel cells in a set of ranges between dark and satu- 
rated, each range spanning twice the brightness values 
of the previous range as will be further described 
below. Counters are used to accumulate the number of 

15 values in each range for a frame period. The corre- 
sponding counts each represent the height of a respec- 
tive bar on a hypothetical histogram which is repre- 
sentative of the data collected. Since the data is 
partitioned into counts, it is also sometimes conve- 

20 nient to consider the counters and the associated 
ranges as bins so the terms counts and associated 
ranges, bins, and histogram bars- will be used 
interchangeably. There are eleven bins for the non- 
zero, non- saturated, two- to-one light level ranges, 

25 one bin for the saturated range, and one bin for the 
zero or dark range- Other ranges besides two- to-one 
may be selected and the bins do not have to cover 
equal light level ratios. The two-to-one difference 
in adjoining bins was chosen mainly because of the 

30 relatively simple decoders needed to enable the counts 
for each of the histogram bars based on the linear, 
binary encoding of the pixel illuminance readings. 
The selection of equal ratios for each of the histo- 
gram bars also has the advantage that such bins are of 
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equal width on a logarithmic plot. Changing the expo- 
sure of the camera changes the readings by equal per- 
centages which shifts them by equal increments on a 
logarithmic scale and thus by equal fractions of a bin 
5 width. Therefore, changes in exposure tend to trans- 
late rather than to materially alter the shape of the 
histogram curve. This is a convenient feature for the 
camera exposure concrol function. In an alternate 
embodiment, che bin ranges are chosen as equal incre- 
10 merits on a linear scale. 

The information from the thirteen counters 
which register the histogram is read by control logic 
84 where it may be analyzed and used as the sensed 
variable in a feedback loop to control the integration 

15 time of image sensor array 60 and to set the optional 
input variable attenuation filter 52 to establish the 
proper camera exposure. The histogram may also be 
used to set the operating characteristic of display 
pixel luminance mapping control 76 so as to dynami- 

20 cally select the function which controls the mapping 
of the camera pixel illuminance reading onto the dis- 
play pixel luminance level for the corresponding dis- 
played pixel. It should be understood here that if 
the mapping of camera pixel cell output to display 

25 pixels is not one-to-one then the functional corre- 
spondence for this mapping would be altered to apply 
to the mapping of the illuminance of an element of the 
scene recorded by the camera to the luminance of the 
corresponding element on the display. 

30 in this illustrative embodiment, the camera 

covers a range of non- saturated and non-z ro readout 
between 1 and 2046, whereas the display has an eight 
bit non-zero, non- saturated range between 1 and 254 
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which is likely to result in even less th^n the 254- 
to-1 range in pixel luminance on the display screen. 
As will be described in more detail below, the camera 
exposure is adjusted so that, when adequate light is 
5 available, the image exposure is generally made as 

high as possible just short of saturating an undesir- 
able high number of the pixel illuminance readings. 
This has the advantage of providing the greatest reso- 
lution for pixels in the readout range of the camera 

10 and also of clipping the often over-bright light lev- 
els from pixels which are saturated. For example, 
enough saturated pixels would normally be tolerated to 
allow the saturation of the very few pixels on which 
the image of the headlamps of a trailing vehicle have 

15 been projected. 

Having established the exposure which tends 
to push the histogram toward the high end of the 
range, the lower extent of the histogram, which 
includes a substantial number of non-zero pixel 

20 readings, is determined- This is- the- lower extent of 
the camera pixel illuminance range which should be 
mapped to the display. If all 2046 camera pixel 
illuminance levels were mapped to the display when the 
scene itself does not posses a very high contrast, the 

25 contrast of the displayed scene would be very low and 
detail would be lost, appearing "washed out" on the 

the highest contrast which spans the range from the 
lower illuminance pixels as determined above to the 
30 highest non-saturated pixel readings should be used. 
The available mapping functions should, in turn, 
include a reasonable range of contrast levels without 
being so extreme on either the high or low contrast 
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ends of the scale so as to yield display images which 
are more difficult rather than easier to interpret. 

There are two basic cases of pixel illumina- 
tion distribution when the light level from scene 24 
is so low that the camera is at its maximum exposure 
level. In the first and generally less common case, 
the pixel illumination distribution has a pattern with 
the majority of pixels lying a significant distance 
from both the saturated and the zero reading ends. of 
the histogram distribution. In this case, mapping 
functions which clip or compress both the highest and 
the lowest camera readings may be used. 

In the second case, the. light level is so 
low that the distribution extends to the low end of. 
the camera sensitivity range. In this case, the upper 
extent of the histogram range which contains an appre- 
ciable number of pixels is determined. This is the 
higher extent of the pixel illuminance range which 
should be mapped to the display. Thus, the available 
mapping function with the highest contrast which spans 
the range from the higher illuminance pixels as deter- 
mined above to the lowest non-saturated pixel readings 
should be used. Such mapping functions are described 
with regards to Figure 5 below. 

The present invention may be used either 
with a black and white camera or with a color camera, 
in which case the encoding should preferably be of the 
type for which the camera pixel illuminance and dis- 
play pixel luminance are indicated by one component of 
the video signal and the color by other components . 
In this form, the processing described above is ap- 
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plied to the illuminance component from th^ camera and 
the color components are preferably left unaltered. 

Note that, where possible with the exemplary 
display, the full eight bit brightness range is used 
5 co show variations of illuminance within the scene. 

Ever, then, the wide dynamic range of the camera must 
often be compressed. It would be unsatisfactory to 
additionally use the eight bit pixel luminance control 
to vary the overall display intensity over the wide 

10 range needed for satisfactory viewing over the huge 
range in ambient light level encountered in driving. 
Thus, thjs primary control of the overall display 
brightness is handled by other methods which may in- 
clude variation in back lighting intensity for a 

15 transmissive display, variation in accelerating 

potential for a cathode ray tube or other form of 
emissive display, or by use of a variable attenuation 
filter 106 for the display. These methods along with 
others may be used separately or in combination in 

20 which case the processor determines the cumulative 

effect and apportions display control signal 102 and 
display attenuation control signal 114 accordingly to 
achieve the required viewing brightness of the 
display. This does not rule out use of pixel 

25 luminance to control the brightness but only 

emphasizes the fact; that most displays do not have the 

the scene luminance range and the overall luminance 
level of the display into the one controlling 
30 mechanism. 



Referring again to Figure 3, el v n data 
selectors 150 through 160 receive the camera pixel 
illuminance r ading as ah 8-bit reading CM[7:0] plus 
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multiplication factor of eight high range indication 
CM_HR and output an 11 -bit value to data bus D(10:0] . 
When CM-HR is not asserted, the long integration time 
is indicated and the 11 data selectors 150 through 160 
5 route CMO to DO, CM1 to Dl, etc. through CM7 to D7 and 
rouce Q's to D8, D9 , and D10 so as to form the 11 -bi- 
value 

Q 0 0 CM 7 CMS CM5 CM4 CM 3 CM2 CM1 CMO . 

When CM_HR is asserted, the short integration 
10 time is indicated and data selectors 100-110 route ones 
co DC, Dl, and D2 and route CMO to D3 , CM1 to D4 , etc. 
chrough CM 7 to D10 so as to form the 11 -bit value 

CM 7 CM6 CM 5 CM4 CM3 CM2 CM1 CMO 111. 

This approximately implements the factor of eight 
15 multiplication so that the signal D[10:0] is an 11-bit 
linear encoding of the pixel illuminance level. Note 
that there are gaps in this range and ones are routed to 
DO, Dl, and D2 for the high range so that the full scale 
value is all ones. 

20 The logic indicated by 162 through 174 imple- 

ments decoding of the values on bus Dtl0:0] . The output 
of logic 162 is asserted when D10 through DO are all 0, 
indicating a dark pixel cell. The output of logic 163 
is asserted when D10 through Dl are 0 and DO is one. 

25 The output of logic 164 is asserted for the two cases 
when D10 through D2 are 0 and Dl is one. The output of 
logic 165, not shown for clarity, is asserted for the 
four cases when D10 through D3 are 0 and D2 is one. 
Logic circuits 166 through 172 are similar, with the 

30 output of logic circuit 172 being assert d for the 512 
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cases when D10 is 0 and D8 and D9 are 1 . The output of 
logic 173 is asserted for the 1023 cases where D10 is 
one and at least one of D9 through DO are 0 . The output 
of logic 174 is asserted when all of D10 through DO are 
5 one, indicating a saturated pixel cell. 

The logic circuits 16 2 through 174 are shown 
co handle a full range of 11-bit input values. Some 
simplification is possible if the gaps in the input 
range are taken into account. Also, depending on the 

10 ' implementation, data selectors 150 through 160 and logic 
162 through 174 may be merged into an optimized logic 
block. In the embodiment shown, the. 11-bit linear value 
on D[10:0] is available and may be routed to other func- 
tions such as display pixel luminance mapping control 7 6 

15 in cases where the expanded linear form is more conve- 
nient than the eight bits plus range select form. 

The results from the decoding logic are used 
to increment counters implementing histogram bins . For 
example, the output of logic 174 is connected to 2 -bit 

20 (divide -by- four) prescale counter 188 so as to enable 
the count of saturated pixels to be incremented by the 
pixel clock, PX_CLK. The overflow of prescale counter 
1.88 is connected to 16 -bit binary up counter 202 so as 
to enable the 16 -bit counter to be incremented by pixel 

25 clock PX_CLK once every four values received by the 
prescale counter. 

The maximum bin count for this illustrative 
embodiment is 76,800, resulting when all values from the 
320 by 240 pixel sensor fall within the same bin. Since 
30 the maximum value in signed 16 -bit arithmetic is 32,767, 
the divide-by-four prescaler limits the maximum counts 
to a reasonable range. Prescaler 188 is preferably 
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preset to binary 11 when counter 202 is reset prior to 
accumulating the count over one frame. In this way, the 
prescaler overflows on and thereby registers the first 
count in the counter 202. This is useful since most of 
the light from a single headlamp may be projected on one 
pixel and a counter with the preferred prescaler will 
"not miss this single count. Buffer 216 is enabled by 
assertion of address signal S12 to transmit the count z 0 
che bus CNT(15:0] for readout by controller 218. 

There are a total of 13 similar counters 190 
through 202. The middle group of six, 193 through 198 
are no: depicted in Figure 3 for clarity. Each counter 
is enabled to count pixels in its respective range and 
each may be read by controller 218. . The respective 
pixel illuminance ranges for counters 160 through "202 
are: 0, 1, 2-3, 4-7, 8-15, 16-31," 32-63, 64-127, 128- 
25S, 156-511, 512-1023, 1024-2046, and 2047. 

A signal from control logic 84 to controller 
218 causes the momentary assertion of a counter initial- 
ization pulse on CLR_CNT to clear the counts and ini- 
tialize the prescalers to register the first count. 
Controller 218 further responds by asserting count 
enable signal CNTJBN for exactly one frame period 
beginning at the first ensuing frame sync pulse and 
ending at the following frame sync pulse on FRM_SYNC. 

Control logic 84 sends an address on A [3:0] 
with a read instruction in response to which controller 
218 responds by asserting the corresponding line for the 
selected, counter on S[12:0] and gating the count re- 
turned on CUT [15:0] to DOUT[15:0] from which it is read 
by control logic 84 . 



WO 99/33634 



PCT/US98/27654 



-32- 

In an optional embodiment, the. function of 
controller 218 may be expanded to additionally use the 
HOR_SYNC signal to count rows in the frame and to 
thereby select bracketed horizontal bands of the frame 
5 over which to record individual histograms. It may be 
particularly advantageous to evaluate a histogram which 
does not include the upper portion of the image which 
may be viewing large portions of the sky. The sky may 
be brighter than the rest of the image, which is most 
10 likely to contain information most useful to operator 
22 . 



Referring now to Figures 2 and 3, horizontal 
sync HOR_SYNC, frame sync FRM_SYNC, pixel clock PX_CLK, 
and the nine signals encoding the pixel illuminance 

15 CM [7:0] and CM_HR as shown in Figure 3 are depicted -by . 
line 68 of Figure 2. Address lines A [3:0], output data 
bus DOUT[15:0], active low write enable NWRT, and image 
brightness detector control select signal BRT_SEL in 
Figure 3 are, in Figure 2, bus 86 which links control 

20 logic 84 to image brightness detector 74 . 

Some cameras output a substantial number of 
non-zero low value readings even when the scene is 
completely dark. Thus, there may be residual counts 
particularly in counter 191 which counts pixels with an 

25 illuminance reading of 1 and perhaps also in counter 192 
which equntis illuminance readings of 2 and 3. These 
dark reading counts should be recorded or esexiSSe^Wd- 
taken into account so that it is not always assumed that 
a substantial number of pixels are illuminated at these 

30 levels during low light conditions. 

The logic described with respect to Figure 3 
may be implem nt d as discrete logic, programmable 
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arrays, custom integrated circuits, software, or any 
combination of the above. Blocks may be combined or 
further divided within the spirit and scope of the 
present invention. 

5 Display Pixel Luminance Mapping Control 

Referring now to Figure 4, an illustrative 
embodiment for implementing display pixel luminance 
mapping control 76 is shown. The same camera system and 
display described with regard to Figure 3 above are used 
10 for the following illustrative example. 

Memory 23 0 in Figure 4 corresponds to the 
display pixel luminance mapping control 76 of Figure 2. 
Address inputs MA12 through MA9 in Figure 4 correspond 
to bus 8 8 from control logic 84 in Figure 2. Address 
15 inputs CM7 through CMO and CM_HR in Figure 4 correspond 
to camera system output signal 68 in Figure 2. Data 
lines DM0 through DM7 in Figure 4 correspond to display 
luminance signal 78 in Figure 2. 

Read only memory 230 is programmed with a 
20 number of lookup tables to translate the illuminance 
portion of camera system output signal 68 into pixel 
luminance signal output 78. Control logic 84 outputs a 
static value on MA9 through MA12 to select a particular 
block of 512 bytes in the memory. The address within 
25 the 512 block is supplied by the pixel illuminance 
reading from camera system 26 in an 8-bit plus range 
select form. This 9-bit number requires 512 bytes 
whereas the 11-bit linearly encoded form requires 2048 
bytes, approximately three quarters of which would not 
30 be access d. 
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Referring now to Figure 5, a graph of display 
luminance mappings is shown. Each of the eleven curves 
shown is a plot of the 8 -bit output value, shown on the 
vertical axis, as a function of the 11-bit input value, 
shown in linear form on the horizontal axis. The 
exemplary plots are as nearly linear as the quancizeci 
encoding will allow. A linear plot on a log-log graph 
has che property that, throughout the illuminance range 
cf che scene, a particular contrast ratio in the scene 
is mapped onto another particular contrast ratio on che 
display . 

Memory 230 has great flexibility to implement 
other curves. In an alternative embodiment, memory 230 
is writable, so that more curves may be supplied and a 
mapping may be constructed based on the analysis of a 
particular scene. 

The points programmed into the memory are 
integer valued outputs which are a function of integer 
values inputs. They are plotted as integer values 
connected by lines to prevent loosing track of the 
curves at: the lower left of the curves where the integer 
points are widely spaced. Also note that when the input 
value with the eight bits plus range selection exceeds 
255, only the values whose three least significant bits 
are binary 111 are used so the actual curves are some- 
what: cotarser fchagt «bqm» for inpijts wfei^h exceed 255. 

The five curves labeled Yl, Y2, Y4, Y8 and Y16 
are used when the scene is bright enough such that 
camera system 26 is set so that the illuminance readings 
are as high as r asonable without having an undesirable 
number of pixels which are at the maximum and presumably 
saturated 2047 illuminance reading. This is the desired 
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criteria for setting the camera sensitivity as explained 
above with regards to Figure 3. Curve Y16\ Y8 , Y4 or Y2 
is selected if an analysis of the histogram shows that 
most of the non-zero pixel readings are above 16, 8, 4 
or 2 respectively. If not, Yl is selected. To select 
the curve, control logic 84 maintains a static address 
value on MA[12:9] which selects the 512-byte block in 
which che required curve is programmed in memory 230 of 
Figure 4 . 

When the majority of pixel illuminance values 
fall between 4 and 511, curve V16 is selected, and when 
che majority of the pixels fall between 2 and 1023, 
curve V4 is selected. 

Under dim lighting conditions with the camera 
at maximum gain and when an appreciable number of pixel 
readings are below 2, one of curve W16 , W8, W4, or W2 is 
selected if the majority of the pixel readings fall 
below 128, 256, 512, or 1024 respectively. Otherwise, 
Yl is selected. 

Note that the number suffix represents the 
relative slope of the curve. Asuffix of 16 represents 
contrast enhancement (signal expansion) , 8 represents 
flat contrast, 4 represents modest contrast loss (mild 
range compression) , and 2 represents relatively pro- 
nounced contrast loss (pronounced range compression) . 
Thus, the curves with the higher number suffixes repre- 
senting the highest applicable contrast are given 
preference in the above selections. 



Although a memory has been used in the illus- 
trative embodiment described with r gards to Figures 4 
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and 5, other forms of hardware or software may be used 
within the scope and spirit of the present invention. 

Referring now to Figure 6, a graph showing the 
control characteristics of input variable attenuation 
5 filter 52 and display variable attenuation filter 106 is 
shown. In a preferred embodiment, filters 52 and 106 
are both solution phase elect rochromic attenuating 
windows. Such windows are constructed using the tech- 
niques described in U.S. Patent 4,902,108 titled "Sin- 

10 gle - Compartment , Self -Erasing, Solution-Phase Electro- 
chromic Devices, Solutions For Use Therein, And Uses 
Thereof" to H. J. Byker and assigned to Gentex which is 
hereby incorporated by reference. Variable attenuating 
filter windows of this type have a chemical layer 

15 sandwiched between two layers of glass, each of which 
has a transparent conductor on the surface which con- 
tacts the chemical. The curve, shown generally as 24 0, 
shows increasing window attenuation along the decreasing 
vertical axis. When the voltage between the opposing 

20 electrodes is low, accomplished by shorting them or 
allowing the chemical to self discharge, the attenuation 
is low. As depicted by curve section 242 in Figure 6, 
when the control voltage applied to the electrodes 
increases from 0 to approximately 0.35 volts, the layer 

25 becomes slightly attenuating. As control voltage is 
increased, the attenuation increasing smoothly with 
t increasina control voltage as shown fey curve section 

244 . A maximum attenuation is achieved when the applied 
control voltage is a little over one volt, as shown in 

30 curve section 246. This characteristic is typical of a 
fairly wide class of solution phase elect rochromic 
devices and reasonably reproducible from one unit to 
another, and reasonably stable with temperature. 
However, temperature compensation may optionally be 
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included in using the voltage setting to control the 
attenuation of the electrochromic window. 

To utilize this voltages verses attenuation 
characteristic, a representation of the characteristic 

5 curve 240 describing the attenuating window being con- 
trolled is programmed into control logic 84. To sec the 
window to a particular attenuation level, control logic 
84 determines the control voltage which corresponds to 
the desired attenuation and sends a command to the 

10 attenuation control circuit to cause this control 
voltage to be applied to the electrochromic attenuator. 
A memory lookup similar to that described with regards 
to Figure 4 above may be used to implement use of the 
attenuation verses voltage characteristic. 

!5 By adjusting chemical composition and layer 

thickness, the maximum attenuation attainable by a 
particular electrochromic attenuator is adjustable over 
a wide range. Increasing the concentration or thickness 
of the electrochromic layer in general increases the 

20 maximum attenuation. In the design, the maximum attenu- 
ation of each electrochromic filter should be adjusted 
to match the maximum attenuation required after balanc- 
ing this with the integration time adjustment for the 
camera or the intensity adjustments for the display. 

25 Also, duty cycling and optical feedback loops are 
alternate devices which may be incorporated in the 
control of the attenuation of the electrochromic fil- 
ters . 



30 



For input variable attenuation filter 52, 
analysis of the histogram is pr ferably used to deter- 
mine the required camera exposure. Control logic 84 
then apportions the exposure between th integration 
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time setting for the camera and the determination of the 
proper transmission for the electrochromic filter. 
Having determined the desired transmission, a voltage is 
selected as described and input attenuation control 
5 signal 94 is sent to input attenuation control 70, which 
generates input attenuation filter signal 72 for input 
variable attenuation filter 52. 

For display variable attenuation filter 106, 
the camera exposure control in combination with image - 

10 brightness detector 74 and display pixel luminance 
mapping control 76 serve to maintain the display at a 
relatively stable luminance level until the scene is so 
dark that the camera can no longer detect large portions 
of the scene. Thus, the function of display brightness 

15 control is primarily to vary the overall brightness to 
match the display brightness to the ambient light 
conditions. The best measurement of the ambient light 
level is obtained from ambient light sensor 34 which is 
positioned to view essentially the same field that the 

20 driver normally sees. This light level is preferably 
subjected to a time average of , for example, 15 seconds 
to derive the stabilized ambient light level used to 
determine the required display brightness. 



Referring now to Figure 7, a graph of display 
25 attenuation ratio as a function of ambient light is 

— *~ ~~ 

that full brightness is the base line and that the 
display will be set to this brightness only for very 
high ambient light conditions. 

30 Curve 260 depicts the display attenuation 

ratio verses ambient light level characteristic on a 
log-log plot. Here an attenuation ratio of one is used 
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to indicate full brightness and, for example, an attenu- 
ation ratio of 10 to indicate that the "display is at 
one -tenth of its full brightness. 

For very bright ambient light conditions, 5000 
lux and above for example, the display is maintained at 
full brightness as indicated by curve section 262. As 
che ambient lighc level decreases over a large range, 
Che display brighcness is attenuated as indicated by- 
sloped curve section 264. Finally, for low ambient 
light levels, below 1 lux for example, the display 
illumination is maintained at a relatively constant 
value as indicated by curve section .266 . 

Curve 260 illustrates a display brightness 
about 50 times lower for very low ambient light condi- 
tions. Specific numbers and threshold values will vary 
considerably with the position of the display and its 
type. It is understood that the numbers which are 
optimal for a given configuration may differ consider- 
ably from those given here and should be established 
after careful evaluation of a particular system. 

Control logic 84 determines the desired 
attenuation of display 96 from full brightness based on 
the ambient light level and a prescription of brightness 
or attenuation such as the one depicted in Figure 7. 
Control , logic 84 then apportions the reductions from 
full brightness between the direct control of the 
display brightness through display control signal 104 
and the setting of display variable attenuation filter 
106 through display attenuation control signal 114 so as 
to achieve the desired display intensity. 
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A vehicle viewing system capable of providing 
an image of a scene to an operator under a wide variety 
of environmental conditions has been presented. While 
the best modes for carrying out the invention have been 
described in detail, those familiar with the art to 
which this invention relates will recognize various 
alternative designs and embodiments for practicing the 
invention as defined by the following claims. 
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Wh^t Is Claimed Is; 

1. A vehicle viewing system for displaying 
an image of a scene to a vehicle operator, the system 
comprising : 

a camera system to generate a signal corre- 
sponding to the scene by integrating light from the 
scene incident on pixel cells, the pixel cells having a 
variable integration time; 

a display system to present the image of the 

scene; and 

a processor system in communication with the 
camera system and the display system, the processor 
system operable to determine the camera system integra- 
tion time based on brightness levels in the scene. 

2. A vehicle viewing system as in Claim 1, 
wherein the camera system comprises: 

an optical sensor; and 

a lens system to focus the image of the scene 
onto the optical sensor. 

3. A vehicle viewing system as in Claim 2, 
the camera system further comprising: 

a variable input attenuation filter to limit 
the light from the image which strikes the optical 
sensor. 

4. A vehicle viewing system as in Claim 2 
wherein the optical sensor is an array of photogate CMOS 
active pixel sensors. 

5. A vehicle viewing system as in Claim 2 
wherein the optical sensor is operative to implement a 
plurality of architectures for integrating and process- 
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ing che image and wherein the processor system is 
operative to determine the measures of the image bright- 
ness and to determine which architectures will be in use 
based on the image brightness measures. 

5 6 . A vehicle viewing system as in Claim 2 

wherein the optical sensor is operative lo reset at 
lease one pixel sensor during an integration period and 
che processor syscem is operative to determine if each 
of the pixel sensors is to be reset, and, for each pixel 
10 sensor to be reset, the reset time during the integra- 
tion period. 

7 . A vehicle viewing system as in Claim 2 
wherein the optical sensor is operative to produce a 
digital signal corresponding to the scene. 

15 8. A vehicle viewing system as in Claim 7 

wherein the digital signal comprises a magnitude and a 
range for each pixel cell, the range indicative of the 
integration period used by the pixel cell. 

9 . A vehicle viewing system as in Claim 3 
20 wherein the variable input attenuation filter comprises 

am elect rochromic window. 

10. A vehicle viewing system as in Claim 1 
u^^^Jwheiaein, ^he^processjar r 9ystem y corner i^sgs an 

ness detector to sample the brightness values of the 
25 image produced by the camera system and to determine 
measures representative of image brightness. 

11. A vehicle viewing system as in Claim 10 
wherein the image brightness detector is operable to 
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V 

determine a histogram of image brightness values for 
image frames output from the camera system. 

12. A vehicle viewing system as in Claim 10 
wherein the processor system further comprises a display 
5 pixel luminance mapping control that uses the image 
brightness measures to determine luminance settings for 
che display system. 

13 . A vehicle viewing system as in Claim 1 
wherein the processor system is operable to select a 
10 curve from a plurality of curves, each curve relating 
display luminance as a function of image brightness 
measures, and wherein the processor system is operable 
to output a display luminance value based on an image 
brightness value using the selected curve. 

15 14 . A vehicle viewing system as in Claim 1, 

wherein the processor system is operable to modify the 
color balance of the display system so as to reduce the 
relative amount of blue light in dim ambient light 
conditions . 

20 15. A vehicle viewing system as in Claim 1, 

wherein the display system comprises: 

a display for emitting light; and 
a display variable attenuation filter for lim- 
iting the light emitted from the display. 

25 16 . A vehicle viewing system as in Claim 15, 

wherein the display variable attenuation filter is an 
electrochromic window. 

17. A vehicle viewing system as in Claim 1 
further comprising: 
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an ambient light sensor for detecting the 
level of ambient light; and 

a control logic for determining display system 
brightness based on the level of ambient light. 

18. A vehicle viewing system as in Claim 1 
further comprising: 

a direct glare sensor for detecting the level 
of glare seen on the display system/ and 

a control logic for determining display syscem 
brightness based on the level of glare seen on the 
display system, 

19. A system for determining brightness 
levels in an image developed by a vehicle viewing 
system, the image composed of a plurality of digitized 
pixel values, the system comprising: 

means for reading each of the plurality of 
digitized pixel values; 

logic for determining into which of a plurali- 
ty of ranges each of the digitized pixel values falls; 

a counter corresponding to each of the plural- 
ity of ranges; and 

a controller in communication with the means 
for reading values, the range determination logic and 
the counters, the controller operative to read a value, 
route the value through the determination logic, incre- 

value . 

20 . A system for determining a luminance 
setting for each of a plurality of display pixels in a 
vehicle viewing system, wherein each display pixel 
represents a corresponding image pixel, and each image 
pixel having a brightness level, th system comprising: 



WO 99/33684 



PCT/LS98/27654 



-45- 

means for determining a histogram from image 
pixel brightness levels; and * 

a logic for determining the luminance for each 
display pixel based on the brightness of the correspond- 
5 ing image pixel and the histogram. 

21. A system for determining luminance 
secc-ngs of Claim 20 wherein the logic for determining 
the luminance for each display pixel is a memory with an 
address determined from the image brightness histogram 
10 and the corresponding image pixel brightness. 



WO 99 33684 



i/6 



PCT/US98/27654 




WO 99/33684 



PCT/L ! S98/-:?654 



2/6 



24 ^ 50 




52 



55^ 50 



65 



62 



INPUT 
VARIABLE 
ATTENUATION 
FILTER 



LENS 
SYSTEM 




IMAGE 
SENSOR 
ARRAY 




IMAGE 






SENSOR 
INTERFACE 




T 


AND CONTROL 
i I 



72 



64 



INPUT 
ATTENUATION 
CONTROL 



AMBIENT 
LIGHT 
SENSOR 



68 



c 96 1 



DIRECT 
GLARE 
SENSOR 



94 

30 



MANUAL 
INTENSITY 
ADJUSTMENT 



116 



36 98 



I 



118 



114 



32 



22 



74 



54 55 

L 



CONTROL 
LOGIC 



IMAGE 
BRIGHTNESS 
DETECTOR 



88 
76 

80 
90 



DISPLAY 
PIXEL 
LUMINANCE 
MAPPING 
CONTROL 



78 



DISPLAY 
ATTENUATION 
CONTROL 



112 



OPERATOR 



108 



110 



^ 106 ^104 



OISPLAY 
VARIABLE 
ATTENUATION 
FILTER 



DISPLAY 
INTERFACE 



702! 
100 

1 



DISPLAY 



82 



92 



0&* 



WO 99/33684 



PCT/US98/17654 



3/6 



CNT(15.Q| 



A(3:0| 



OOUT (15:01 




NWRT 


O 


BRT.SEL 


O 


HOR.SYMC 


O 

O 


FRM.SYWC 



, PX.CLK 



218 



S[12:0j 
CNT_EM 



CM_HR 



CM|7:0l 



fit 
fit 



160 



159 



fit 
fit 



755 



757 



fit 



755 

755 



•753 



fit 



752 



757 



fit 



750 



GND 



774 



/S5 



00 
Ql 

CE CEO 
>C TCU 



n 



0( 15 0) 
CE CEO 

> c ci* TC 



202 



[187 



00 1_ 

I 01 
CE CEO 
>C TC 



Of 15 0) 
CE CEO 



Z 



201 



186 



171 



164 



\ 163 



R 



752 



755 



77* 



oo 

01 

CE CEO 
>C TC 



Q( IS 0) — 
CE CEO — 

r CLR 



7 



200 



M qi - 

CE CEO ICE CEO — 

»C TcP^>C aR Tc|- 



Q< 15 0) 

LI 

I 



199 



CE CEO 
>C TC 



00 - 
01 



0(15 0) 
CE CEOl- 
C TC 



z 



192 



177 



CE CEO 



oo - 

01 



0(15 0) 
CE CEO 



797 



176 



CE CEO 
>C TC 



00 - 

01 - 



rt 



0(15 0) 

a ceo 

zr 



0(10:0] 



z 



190 



275 



•274 



•273 



206 



-205' 



204 



WO 99/33484 PCT/US98/27654 

4/6 



vcc 

o 



230 



MA[12:9] 



CM[7:0] 



MA12 
MA11 
MA10 
MA9 
CM_HR 
CM7 
CM6 
CM5 
CM4 
CM3 
CM2 
CM1 
CMO 



VPP PGM 



A12 

A11 

A10 

A9 

A8 

A7 

A6 

AS 

A4 

A3 

A2 

A1 

AO 



D7 
D6 
05 
04 
03 
02 
01 
00 



^ Of 

u 



0M7 
DM6 
0M5 
0M4 
DM3 
DM2 
DM1 
DMO 



DM(7:0] 



GND 



WO 99/33684 



PCT7US98/276S4 




WO 99/33*84 



PCT/XS98/77654 




1 



THIS PAGE BLANK (uspto> 



:**•-.... fcft^.-..v,.<3g 



PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




(51) Internal nal Patent Classification * : 


A3 


(11) International Publication Number: 


WO 99/33684 


H04N 5/235, 9/73, 5/57 


(43) International Publication Date: 


8 July 1999 (08.07.99) 



(21) International Application Number: PCT/US98/27654 

(22) International Filing Date: 30 December 1998 (30.1 2.98) 



(30) Priority Data: 

09/001.855 



31 December 1997 (31.12.97) US 



(71) Applicant: GENTEX CORPORATION [US/US]; 600 North 
' Centennial. Zeeland. MI 49464 (US). 

(72) Inventors: BECHTEL. Jon. H.; 136 Sunrise Drive. Holland. 

MI 49423 (US). STAM. Joseph. S.; 345 South LaJceshore 
Drive. Holland. MI 49424 (US). FOSSUM. Eric R.; 5556 
Pinecone Road. La Crescent*, CA 91214 (US). KEMENY 
Sabrina. E-; 5556 Pinecone Road. La Crescenta. CA 91214 
(US). 



suznated States: al, am. n. i , «<j. --• 
bTca. CH. CN, CU. CZ. DE. DK. EE. ES. FI. GB. GD. 
GE GH. GM. HR, HU. ID, IL. IN. IS, JP. ^ KG KP . 
KR KZ. LC. LK. LR. LS. LT, LU. LV. MD, MG MK, 
MR MW, MX. NO. NZ. PL PT. RO RU. SU SE. SC. 
SI SK. SL. TJ. TM. TR. TT. UA. UG, UZ. VN. YU. ZW. 
ARIPO patent (GH, GM. KE. LS. MW, SD, SZ. UG. ZW). 
EutSpIS, (AM. Ai BY. KG. KZ. MD, RU. TJ TM). 
European patent (AT. BE CH. CY. DE DK ES. FI. FR. 
GB. OR. IE IT. LU. MC. NL. PT. SE). OAPI patent (BF 
BJ. CF. CG. CI, CM. GA. ON. GW. ML. MR. NE. SN. 
TD. TG). 



Published 

With international search report. 

Before the expiration of the time limit for amending the claims 
and to be republished in the event of the receipt of amendments. 



(74) Agents: CHUEY. Mark, D. et al.; Brooks & Kushman. 22nd 0 f publication of the international search report: 

floor. 1000 Town Center. Southfield. MI 48075 (US). ua,e puo 23 September 1999 (23 09.99) 



(54) Tide: VEHICLE VISION SYSTEM 
(57) Abstract 

A vehicle viewing system including a camera 
system for generating a signal corresponding e>a scene 
by integrating light from the icge incident ooptwt 
cells having a variable i taeg u iio n *5^2 
system for presenting a visual representati on of tne 
scene, and i processor system overawe to «wminc 
the camera system integranoo tiroe baaed on brightness 
levels in the scene. The camera s^tOTprcfaaMy 
includes an optical amy of legate CMOS active 

pixel sensor cell, and ^^"C 
a plurality of integration and processing srchittctures- 

The camera ^.^JffS 
and an input attenuate, iter to limit UgtattOdng 
the optical amy. The processor system includes an 
image brightness detector » determine m easures ot 
overall image brightness and a display p»el hauiuance 
mapping control to determine luminance settingl tor 
the display system. The image brightness «tertorinay 
include circuitry that produces a bnghtr^ histogram. 
The proc esso r system may determine the tnttasny ot 
the display system based on the brightness of the 
scene, ambient light levels, and glare onthe dlgay. 
The display system includes a display and a display 
attenuation filter for limiting the intensity as viewed 
by the operator. 
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